Fluoroscopic registration system and method

ABSTRACT

A method and system for registering volumetric scan data of a patient surgical site with actual patient position are disclosed. In practicing the method, position data relating to (i) the x,y,z coordinates of a detector screen in fixed coordinate system, and the x,y, coordinates of patient features on the screen are used to determine the x,y,z coordinates of the patient features in the fixed coordinate system. These coordinates are then matched with the coordinates of the same patient features taken from pre-op scan data, e.g., CT scan data, to place the CT scan data in the fixed coordinate system.

This patent application claims priority to U.S. Provisional PatentApplication Ser. No. 60/230,633 filed on Sep. 7, 2000, which isincorporated in its entirety herein by reference.

FIELD OF THE INVENTION

This invention relates to image-guided surgery, and in particular, toregistration and fluoroscopic imaging methods.

BACKGROUND OF THE INVENTION

In a typical image-guided surgical procedure, pre-operative scan data,e.g., CT or MRI scan data, of a patient region is obtained. This is doneconventionally by immobilizing a patient during a scan with a CT or MRIimaging machine. The patient is then moved to a surgical setting fortreatment of the target region. During the surgical procedure, the scandata is used to reconstruct hidden or subsurface images of the patienttarget region, to guide or assist the surgeon in the surgical procedure.For example, the scan data may be used to reconstruct a subsurface imageof the target region as seen from the position and orientation of asurgical instrument being held by the surgeon.

It is important, of course, for the coordinates of the scan-data imagesbeing shown to the surgeon to closely match the coordinates of theactual target region. For this reason, it is important to calibrate thescan data with the actual position of the patient, keeping in mind thatthe scan data is obtained in one location, and the patient is moved toanother location for the surgical procedure. Where the target region isa patient's head region, this coordinate matching may be done by placingfixed-position fiducials on the patient's head during the scanningprocedure, and retaining the fiducials at the same head positions forthe surgical operation. Coordinate matching can then be done by aligningthe positions of the fiducials in both reference frames. This approachrelies on the fact that the fiducials placed on rigid, non-deformablestructure (e.g., skull) are retained in substantially the same position(patient coordinates) for both the scanning and surgical procedures.

For other surgical regions, e.g., the spine, it is virtually impossibleto retain fiducials at fixed patient coordinates between the scanningand surgical operations. In this case, it is necessary to recalibratethe positions of the fiducials in the patient coordinate system everytime the patient moves or is moved. Ideally, the surgeon would want toforego the use of fiducials altogether, since their placement involvesadditional inconvenience to the patient and surgeon.

It would therefore be useful to provide an improved system forregistering pre-op scan data in a patient coordinate system during asurgical operation, to be able to use the scan data to accuratelyconstruct subsurface image data corresponding to the actual patientcoordinate system.

Also during a surgical operation, a surgeon will often wish to check theexact placement of surgical cut, or placement of a tool or implant atthe patient site. This can be done conventionally, by taking afluoroscopic image of the patient region, and displaying the 2-D imageto the surgeon. Unfortunately, the surgeon will be limited in the numberor times and duration of fluoroscopic imaging, due to the need to limitthe amount of x-ray exposure to both the patient and medical personnel.

It would therefore be desirable to provide a method and system ofvirtual fluoroscopy that allows a surgeon to view a patient region“fluoroscopically” from a number of different angles, and over extendedperiods of view, without actually exposing the patient and others in thesurgical theatre to exposure to x-rays.

In particular, it is would be useful to construct virtual fluoroscopicimages in real time, e.g., without significant computation time.

SUMMARY OF THE INVENTION

The invention includes, in one aspect, a method of registeringvolumetric scan data of a patient surgical site with actual patientposition. The method includes the steps of (a) obtaining, at a scanstation, volumetric scan data of a surgical site of a patient, includingdata relating to plural selected features in the region, and storing thescan data in digital form in a scan-data file, (b) after moving thepatient from the scan station to a surgical station, positioning anx-ray source and a detector screen for imaging at least a portion ofsaid target region fluoroscopically at a selected orientation, (c)determining the coordinates of the x-ray source and detector at thepositions in a fixed coordinate system, (d) activating the x-ray sourceto produce a fluoroscopic image of the patient region on the detector,(e) determining the image coordinates of the selected features in thetarget region in the fluoroscopic image in the fixed coordinate system,(f) using the coordinates determined in (c) and (e) to determine theactual coordinates of the selected features in the fixed coordinatesystem, and (g) matching the coordinates determined in (f) with those ofthe same features in the scan data, to place the scan data in the frameof reference of the patient.

The method may further include using the scan data to construct asubsurface image of the patient, as viewed from a selected position andorientation in the surgical station. Where the selected position andorientation is with respect to the tip of a surgical instrument, themethod further includes tracking the position of said instrument in thesurgical station.

In one embodiment, the plural selected features include at least threefeatures, and steps (b)-(e) are carried out at different selectedpositions of the x-ray source and detector screen.

In another aspect, the invention includes a system for registeringvolumetric scan data of a surgical site with actual patient position.The system includes (a) a scan-date file for storing, in digital form,volumetric scan data of a surgical site of a patient, including datarelating to plural selected features in the region, (b) an x-ray sourceand a detector screen for imaging at least a portion of the targetregion fluoroscopically at a selected orientation, (c) a fixed-positionsensor for determining the coordinates of the x-ray source and detectorat the positions in a fixed coordinate system, and (d) a computationaldevice operatively connected to the data file, detector screen, andfixed-position sensor, for (i) determining the actual coordinates ofselected features of the surgical site in the fixed coordinate systemfrom the determined coordinates of such features on the detector screenand from the coordinates of the screen, and (ii) matching the actualcoordinates of the patient features with those of the same features inthe scan data, to place the scan data in the frame of reference of thepatient.

These and other objects and features of the invention will become morefully apparent when the following detailed of the invention is read inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an arrangement of components of a surgical imagingsystem and a patient in a surgical setting containing the system;

FIG. 2 is a flow diagram of steps executed by a computational device inthe system, for registering pre-op scan data with actual patientposition, and for using the scan data to construct target images from aselected position and orientation;

FIG. 3 shows the projection of rays through a target region onto avirtual array from a virtual source point, in the method of theinvention for constructing virtual fluoroscopic images;

FIG. 4 illustrates the use of voxel elements in calculating pixeldensity values, in constructing a virtual fluoroscopic image inaccordance with the invention;

FIG. 5 shows four registers associated with each pixel element in adisplay screen employed in the present invention;

FIG. 6 shows one assignment of density-value digits to display-screenregisters in the method of the invention;

FIG. 7 is a flow diagram of the fast mapping method in accordance withthe invention;

FIG. 8 illustrates the virtual fluoroscopy method of the invention,showing a virtual fluoroscopic source at two selected points;

FIGS. 9A and 9B show views of a patient spinal column as generated byvirtual fluoroscopy from the two virtual points shown in FIG. 8; and

FIG. 10 is a flow diagram of the steps in the virtual fluoroscopy methodof the invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is designed to enhance several aspects ofimage-guided surgery. In each case, the invention depends on or usespre-operative volumetric scan data of a patient target region.Typically, the scan data acquired are computerized tomography (CT),magnetic resonance imaging (MRI), or ultrasound scan data, obtainedaccording to well-known methods. The scan data are conventionally storedin digitized form in a scan-data file, for later use in the invention.For purposes of the present invention, the scan data have threeimportant characteristics.

First, the target region (volume) that is scanned is divided into athree-dimensional voxel space, where each voxel (small volume element)typically represents an x-y element in the “plane” of scans taken alongthe z direction. (Voxels defined by other types of coordinate systems,e.g., polar coordinates, may also be employed). Second, each voxel isassociated with an M-bit density value that characterizes the scandensity observed for that volume element in space. Third, the voxelcoordinates are placed in a patient coordinate system keyed to one ormore of the following: (i) patient structural features, such as bonefeatures or external surface features, (ii) fiducials, typically smallmetal markers, implanted at selected positions within the patient targetregion, (iii) fiducials placed at fixed positions on the surface of apatient, or (iv) sensor elements placed at fixed positions in or on thepatient.

During a surgical procedure—that is, after the patient has been movedfrom a scanning station to a surgical station—the scan data must beplaced in the new patient coordinate system. This is done, in general,by determining the coordinates of patient physical structure, e.g., bonefeatures or fiducials, and transforming the coordinates of the scan-datavoxels to the new coordinate system.

Various approaches to placing pre-op scan data in registration with thepatient surgical position have been proposed heretofore. In one,fiducials whose position in space can be determined by a fixed-positionsensing unit are placed on the patient. The fiducial positions in thesurgical setting can then be determined directly by the sensing unit,and used as reference points for the scan data, which includes images ofthe fiducials at known voxel positions. The limitation with this method,at least in spinal surgery, is that the fiducials must be placed on thepatient surface, and typically do not provide accurate coordinates forsubsurface structure, e.g., vertebrae.

In a second approach, a fluoroscopic image of the patient target regionis produced on a display screen, and this image is then position-matchedwith a digitally reconstructed radiograph (DRR) produced byreconstructing scan data. By manipulating the coordinates of the scandata until the DRR and actual fluoroscopic image match, scan datacoordinates corresponding to actual patient coordinates are obtained.This method is described, for example, in U.S. Pat. Nos. 4,791,934 and5,901,199. One aspect of the present invention, described in Section IIbelow, is a fast-mapping algorithm that significantly reduces the timeneeded to generate virtual radiographic images, and thus significantlyimproves this image-matching method.

A third approach, which forms another aspect of the present invention,is described in Section I below.

I. Registering Pre-Op Scanning Data with Patient Position

FIG. 1 illustrates components of a system 10 of the invention, shown inoperative relationship to a patient 12 placed face-down on a surgicaltable 14. For purposes of illustrating the describing the invention, thepatient region of interest—the surgical target region—is the patient'sspine, indicated at 16. Although the invention is applicable to anypatient target region, the invention is particularly applicable tospinal region, both because the vertebrae of the spine offer a goodfluoroscopic target (bone), and because the position of the spine isdifficult to track by external markers when the patient moves or ismoved. The same applies to any piece-wise rigid structure (e.g., anypart of skeletal anatomy).

System 10 includes a data-scan file 18 which stores pre-op scan data,typically obtained at a scanning station (not shown) which is differentfrom the surgical station shown. As indicated above, the scan-dataconsists of voxel coordinates, corresponding scan-data, e.g., densityvalues, for each voxel, and coordinates for registration features, e.g.,selected bone features or fiducials. The scan data file is operativelyconnected to a computational device 20, e.g., computer or processor,which carries out the data processing operations of the system, to bedescribed. Also connected to the computer are a fixed-position sensingunit 22 and a display device 24. The display device is preferablyincludes a conventional computer color-monitor screen composed of aframebuffer with an XY array of pixels, where each pixel has four N-bitregisters, e.g., 8-bit registers, corresponding to red, green, blue andalpha registers.

Sensing unit 22 is part of a position tracking system, which ispreferably an optical tracking system (hereafter “OTS”) having sensingunit 22 mounted overhead in view of the operating table scene, andvarious movable objects, each equipped with at least two light emittingdiodes (LED's) mounted on the object. In the embodiment shown, themovable objects include first and second detecting plates 30, 32designed to be placed at selected positions adjacent the patient, asshown, and an x-ray source 38 also designed to be placed at selectedposition with respect to the patient. Another movable object that may beincluded in the system, but which is not necessarily used forregistration, is a surgical instrument 26 designed to be used by thephysician during a surgical procedure.

As shown, X-ray source 38 is equipped with LED's 40; screen 30, withLEDs 34; screen 32, with LEDs 36, and instrument 26, with LEDs 28. TheLED's attached to the movable objects are preferably designed to emitstreams of pulsed infrared signals which are sensed by a plurality ofinfrared detectors (not shown) contained in the sensing unit. Thesensing unit and movable objects are connected to the computer, whichcontrols the timing and synchronization of the pulse emissions by theLED's and the recording and processing of the infrared signals receivedby the detectors. The OTS further includes software for processing thesesignals to generate data indicating the location and orientation of themovable objects in the frame of reference of surgical station. The OTSmay generate the position data on a real-time basis, so that as thesurgical instrument is moved, for example, its position and orientationare continually tracked and recorded by the sensing unit, and when thedisplay screens or x-ray source are moved, their positions in thecoordinate system of the surgical station are known.

The OTS is preferably of the type known as the “FlashPoint 3-D OpticalLocalizer”, which is commercially available from Image GuidedTechnologies of Boulder, Colo., similar to the systems described in U.S.Pat. Nos. 5,617,857 and 5,622,170. However the invention is not limitedto this particular OTS, and other position tracking systems, such assonic position detecting systems, may also be utilized.

X-ray source 38 may be, for example, any of a number of commerciallyavailable fluoroscopic sources such as an OES 9600 obtained from GeneralElectric Medical Systems (Milwaukee, Wis.). The source is typicallymounted on a suitable frame (not shown) for positioning at a selected“view” position with respect to the patient. The detecting plates maybe, for example, conventional rectangular arrays of CCD devices or otherelectronic components that convert radiation intensity intocorresponding electronic signals. The CCD elements in the plates areoperatively connected to corresponding pixel elements in the displayscreen, for displaying the fluoroscopic or shadowgraph image of thepatient region projected onto the detecting plate. For example, if theCCD devices output an 8-bit digital signal, this output can be suppliedto one of the four display screen registers to display the CCD arrayoutput on the display screen. Also as indicated in the figure, the CCDelements making up the arrays in each detecting screen are connected tocomputer 20, for supplying the computer with digitized fluoroscopic datarecorded at the array positions on the plate.

Ultimately, and as described further below, the system will be used todisplay patient target-region images reconstructed from the scan dataand as seen, for example, from the position and orientation ofinstrument 26, for guiding the physician during a surgical operation.

The registration method of the invention is illustrated in FIG. 1, andthe steps in practicing the method, including those steps executed bycomputer 20, are given in the flow diagram in FIG. 2. Initially, source38 and plate 30 are positioned at selected positions to image thepatient target region, e.g., spinal region, from a first selectedposition, as illustrated in FIG. 1. With the source and plate sopositioned (the source position is indicated at P₁) the OTS is activatedto record the positions of both source and plate in the frame ofreference of the sensing unit, that is, with respect to a fixed point ofreference within the surgical station. The X-ray source is thenactivated to produce a fluoroscopic image or shadowgraph of the patientregion, as detected at the detector plate. The array signal intensitydata is supplied to display 24, for displaying the fluoroscopic image tothe user, and also input into a dedicated image file in computer 20.

At this point, the system (with or without the user's assistance)identifies three or more patient-region features that are alsorecognizable in the pre-op scan data. The features may be, for example,radio-opaque fiducials, which can be identified by their high-densityvalues, or bone features that can be identified by their distinctiveshape or position. The user may specify these features by “clicking” onthese features in the image displayed in device 24, or the features maybe “recognized” by the system automatically according to distinctivedensity values, e.g., by sharp density-edge effects. Once the featureshave been identified, the system records the XY positions of thefeatures on the display screen, corresponding to the XY positions on thedetector plate.

The source and detector plate are now shifted to second selectedpositions for viewing the target region from a second angle. In theembodiment shown, source 38 is moved from P₁ to P₂, and the detectorplate (which may be the same plate 30 or a second plate 32) is moved tothe position shown for plate 32. A second fluoroscopic image of thetarget region is then taken, with the image being displayed to the userat device 24 and stored in the processor as above. Again the system oruser identifies the same target features identified earlier, and thepositions of these features on the detector plate are stored.

The program flow in FIG. 2 shows how the system determines the actualspatial coordinates of the patient features, e.g., features a, b in FIG.1. Initially the spatial coordinates of the detector plate(s) at bothviews are determined by the OTS and stored in the computer, as at 42.Source 38 is then positioned at selected position P₁, as at 44, and itscoordinates determined by the OTS, as at 46. The source is activated toproduce a fluoroscopic image or shadowgraph on detector plate 30. Thepositions of features a, b in the shadowgraph are indicated at F_(1a)and F_(1b) in FIG. 1, corresponding to positions x_(1a),y_(ia) andx_(1b),y_(1b) on the detector plate array, and to pixels x′_(1a),y′_(ia)and x′_(1b),y′_(1b) on the display screen. From the known x,y positionsof features a, b on plate 30, and from the known position of the plate,the computer determines the coordinates of F_(1a) and F_(1b)(x_(1a),y_(ia) and x_(1b),y_(1b)) by standard geometric calculations, asat 48, and these coordinates are stored in the computer for later use.

Similarly, after moving the source to selected position P₂, as at 50,and determining the new source position, as at 52, and with detectorplate 32 at a second selected plate position, the user takes a secondfluoroscopic image of the patient region from this second selected view.The second image contains the features of interest, e.g., a, b, asindicated at F_(2a) and F_(2b) in FIG. 1. As above, from the knownpositions of these features on the detector plate, and from the OTSdetermined position of the plate, the coordinates of F_(2a) and F_(2b)are determined, as at 54.

The program now has the x,y,z spatial coordinates for the two sourcepositions P₁ and P₂, and for the plate coordinates of the patientfeatures on both plates (or one plate at two selected position), e.g.,F_(1a) and F_(1b) and F_(2a) and F_(2b). These coordinates allow theprogram to calculate, by standard “intersection” equations, the x,y,zcoordinates of selected patient features, e.g., a, b, as at 56.

The system next determines the scan-data coordinates for the samepatient features in the scan data file 18. This may be done, in oneembodiment, by reconstructing a virtual image from the scan data, andhaving the user “click on” the feature(s), e.g., implanted fiducials ordistinctive bone features, used as landmarks in the fluoroscopic images,e.g., features a, b. Alternatively, the program may automaticallyidentify the landmark features by distinctive density values or densitypatterns, as above.

Once the selected features or landmarks in the scan data are identified,the program will construct a transform that transforms the internalcoordinates of these features into the spatial coordinates of thefeatures determined from the fluoroscopic images, as at 58. Thistransform, when applied to all of the scan-data coordinates, effectivelyplaces the scan-data in the actual frame of reference of the patient.That is, the scan data is placed in registration with actual patientposition.

After executing this transformation, the system can use the scan data toconstruct selected subsurface views, e.g., perspective views) of thepatient target region, corresponding to the actual patient position.These images may be presented to the physician at display device 24. Inone preferred embodiment, the OTS in the system is used to track thereal-time position of a surgical tool or instrument, e.g., instrument26, and the reconstructed images are be displayed from the position andorientation of the instrument, allowing the user to view hidden tissueand the instrument is moved toward or within the patient target site.Methods and algorithms for producing such images, particularly as seenfrom the point of view of a movable instrument, are described, forexample, in PCT application WO 9900052 A1 for Image generation of threedimensional object, which is incorporated herein by reference.

II. Fast 3-D to 2-D Mapping Method and System

In another aspect, the invention provides a rapid method for mapping 3-Dscan data onto a 2-D display format, particularly for use in generatingdigitally reconstructed radiographs (DDRs) or virtual shadowgraphs fromCT, MRI or ultrasound volumetric scan data. Such radiographs are alsoreferred to herein as virtual fluoroscopic images or virtualshadowgraphs, meaning a displayed image that resembles a fluoroscopicimage taken from a selected view, but which is constructed from pre-opscan data rather than actual x-irradiation of the patient site.

The system for carrying out the invention generally includes thecomponents shown in FIG. 1, in particular, a scan data file 18,computational device 20, and display device 24. As described above, thescan file consists of voxel coordinates, corresponding scan-data, e.g.,density values, for each voxel, and coordinates for registrationfeatures, e.g., selected bone features or fiducials. The scan data fileis operatively connected to the computational device, e.g., computer orprocessor, which carries out the data processing operations of thesystem to be described.

The display device is preferably a conventional computer color monitorhaving a display screen composed of an XY array of pixels, e.g., a1280×1024 pixel array. The framebuffer provides, for each pixel, aplurality of N-bit registers, typically four 8-bit registerscorresponding to blue, red, green, and alpha single-valued pixel inputvalues. Each pixel register functions to accept an N-bit color value,for outputting to the corresponding pixel, an electronic signalindicating the corresponding brightness of that color in that pixel.According to an important requirement of the invention, each register ishardwired with an “add” function that allows the register to addsuccessive values (with N or fewer bits) supplied to the register.Virtually all commercially available color monitors have these featuresand are therefore suitable for use in the invention.

In the present system, the computer is connected to the monitor toprovide input to all of the pixels, and to each of the registers in eachpixel element, simultaneously, to allow parallel input to each pixelduring the operation of the system. The operation of the computer, andthe computer software or hardwired program required in the system in thesystem will be appreciated below from the description of the methodcarried out by the system, with reference to FIGS. 3-7.

It is assumed in the following discussion that it is desired toconstruct a virtual fluoroscopic image of a spinal target region 16 inpatient 12, and that the view of the target tissue is that which wouldbe produced by projecting x-rays from a selected source point P₁ onto adetector screen placed at the position occupied in the figure by avirtual detector plate, indicated at 64. The virtual plate is consideredas an array of points, such as points 64, 66, corresponding to the CCDelements in an actual detector plate array, and each point is designatedby a unique x,y coordinate in the array. That is, all points in thearray can be represented by coordinates x_(i),y_(k), i=1 to m, and j=1in an m×n array. Further, the points correspond, through a one-to-onemapping, to the pixels in the XY pixel array of the display device.

In practicing the method, the user selects a desired source point P₁from which the patient target tissue will be viewed, as at 76 in FIG. 7.This may be done, for example, by providing a movable frame-mountedpointer that can be moved to a selected point with respect to thepatient, allowing the field of view angle to be selected according tothe distance between and pointer and the patient, and the vieworientation to be selected according to the orientation of the pointer.The position and orientation of the pointer may be tracked by the aboveOTS (or comparable tracking device) to indicate to the computer theactual coordinates of the virtual source point P₁. In particular, thepointer can be rigidly fixed to the exterior of the real fluoroscopiccamera to allow for natural simulation of its use. Similarly, it can behand-held to permit computing views that would not be possible tophysically realize with a real fluoroscope (e.g., axial views).

With virtual source point P1 selected, the computer constructs a set ofrays as at 78 in FIG. 7. FIGS. 3 and 4 show exemplary rays r₁ and r₂ insuch a construction. The computer selects the coordinates of the virtualdetector points so that at least a portion of the rays pass through thetarget region of interest, as shown in FIG. 3. Preferably thecross-section of the rays encompasses the entire target region, and acentral ray is normal to the plane of virtual plate.

In constructing the virtual fluoroscopic image, the program operates tosum the scan-data density values along each ray in the set of rays, andplace this sum (after scaling) in one of the registers, e.g., thegray-scale register of the corresponding pixel in the display device, todisplay the virtual fluoroscopic image to the user. In order to performthis summing operation, the program must define a frame of reference forthe scan data, so that each voxel in the scan data can be assigned toone or more rays projecting through the patient target region. If thescan data has been placed in registry with the patient position, asdescribed above, the coordinates of the scan data voxels can be assignedactual patient coordinates in the surgical setting. If no suchregistration is performed, or if the patient moves after theregistration operation, the voxel coordinates may correspondapproximately, but not precisely, to actual patient coordinates.

FIG. 4 illustrates the assignment of voxels to rays r1 and r2, asrepresentative of all of the rays considered in the method. The computerknows the actual x,y,z coordinates of P1, and has assigned x,ycoordinates to each of the points in the virtual detector array. (The zaxis is considered here as the axis normal to the plane of the virtualdetector array). Therefore, the x,y position of each ray at each scanslice (along the z axis) is known and assigned a voxel corresponding tothe x,y,z coordinates of the ray at that slice position (z coordinate).As noted above, the voxel coordinates may be actual patient coordinates,or may be coordinates assigned when the scan data was obtained. For somerays, such as ray r₁, the ray may extend through a single column ofvoxels in the patient target region, such as the voxels identified inFIG. 4 as V1p, where p=1 to w; for others, such as ray r₂, the ray mayextend through two or more voxel columns, such as the columns identifiedby voxels V_(2p)p (p=1 to y) and the adjacent voxel column made up ofvoxels V_(3q), q=1 to x). Alternatively, the voxels may be assignedpolar coordinates, such that each ray passes through a single column ofvoxels.

Each of the voxels is associated with a tissue-density valuerepresenting the tissue density seen by the scanning device at thatvoxel position. The density value is scaled, if necessary, to a selectedM-bit number, preferably an 8-bit number to represent, for example, 255different density values. The program then operates to sum the densityvalues along each ray, to produce a summed value representing the totaltissue density that an x-ray would encounter on traversing theassociated column(s) of voxels in the patient target region.

The summing operation along the rays is carried out in parallel for allrays, whereby the total summing operation time is no greater than thatrequired for a single ray. According to an important feature of theinvention, the summing operation is further accelerated by takingadvantage of the ability of the pixel registers to sum values applied tothe registers. How this feature is exploited in the present invention isillustrated in FIGS. 5 and 6. FIG. 5 shows a portion of a display screen70 having an XY array of pixels, such as pixels 72, 74 corresponding tothe XY array of points in the virtual detector array in FIGS. 3 and 4.Each pixel, such as pixel 72, receives input from four N-bit registers,such as registers 72 a, 72 b, 72 c, and 72 d associated with pixel 72and registers 74 a, 74 b, 74 c, and 74 d associated with pixel 74. Inthe normal color-display mode of the display device, the four registersoperate to signal impart a particular color to each pixel, depending onthe relative values supplied to the pixel by the four registers.

Typically, the pixel registers are 8-bit registers, and thereforecapable of holding up to 255 different digitized values. In thediscussion below, 8-bit registers will be assumed, although it will berecognized that the summing operation could be applied to largerregisters, e.g., 16-bit registers.

For each ray, the summing operation is carried out by first consideringthe 8-digit density value as four separate 1-, 2-, or 3-bit numbers. Thereason for splitting the input values this way is to reduce thepossibility that any of the individual registers will overflow. The foursmaller values, representing particular bit positions in the 8-bitnumber, are then assigned to the four color registers. For example, inthe assignments illustrated in FIG. 6, the 8-bit density-value number isbroken down to four two-bit numbers, such that the 8-bit numbern₇n₆n₅n₄n₃n₂n₁n₀ becomes n₇n₆, n₅n₄, n₃n₂, and n₁n₀, with each of the2-bit numbers being assigned to one of the four registers. Anotherpreferred assignment considers the 8-bit number in two groups or threedigits and two groups of one digit, such that the 8-bit numbern₇n₆n₅n₄n₃n₂n₁n₀ become n₇n₆n₅, n₄n₃n₂, n₁, and n₀, again with the foursmaller numbers being assigned to the four pixel registers. An advantageof the later bit distribution is that lowest-bit numbers, e.g., n₁, andn₀, are expected to include higher noise values, and thus may overflowthe associated registers faster during a summing operation, than do thehigher-bit numbers. By limiting the lower-bit numbers to one bitpositions, the tendency to overflow the associated registers isminimized.

In a summing operation, illustrated in FIG. 6, the first voxel 8-bitnumber in each ray (n=1) is read, as at 80 in FIG. 7, and is placed inthe four registers of the corresponding display pixel, in accordancewith the above bit assignments, such that each register receives a 1-,2-, or 3-bit number, as at 82 and 84 in FIG. 7. The program thenproceeds to the next voxel in each ray (n=n+1), as at 86, and repeatsthe above steps by adding the corresponding 1-, 2-, or 3-bit number tothe same registers. As noted above, the registers are hardwired toperform number addition, so the addition operation occurring at all ofthe registers can take place fill rates of 1 Gpixels/second. At eachaddition step, and for each ray in parallel, the program adds the next8-bit voxel value to the associated four registers, and this process iscontinued until all voxels along all rays have been considered, as at88.

To obtain the final density value for each pixel, the values of the fourregisters are summed, preserving the order of the original assignment ofdigits, as at 90 in FIG. 7. That is, any carry over from the registercontaining the lowest-value digits is added to the register containingthe next-higher-value bits, which is added to the register with thenext-higher-value bits and so forth. The final summed value, which willtypically be a 24-32 bit number, may then be scaled to an 8-bit number,as at 92 in FIG. 7, and placed in the screen's gray-scale registers, todisplay the virtual image in gray scale, as at 94 in FIG. 7.

From the foregoing, various advantages and features of the invention canbe appreciated. The mapping method, because it utilizes paralleladdition in a slice-by-slice manner along the scanned tissue volume, andtakes advantage of the hardwired addition feature of the display screenregisters, is approximately two orders of magnitude faster than would beachieved by treating the four color registers in each pixel as a 32-bitregister, and successively summing 8-bit numbers in this expandedregister. This is due to inherent parallelism of the method (all raysare operated on simultaneously) and straightforward implementation onoff-the-shelf hardware. More realistic light transport models (e.g.,involving X-ray scattering) can also be incorporated in this model.

The speed of operation allows for virtual fluoroscopic image generationin real time, in accordance with the method described in Section IIIbelow. For example, accumulating 512 values along each ray on a 512²viewport requires 512³ or 128 MegaAdditions, hence a graphics adapterwith 1 Gpixel fill rate can compute 8 DRRs in a single second. It alsofacilitates other digitally reconstructed radiograph (DDR) applications,such as the one described in the background of Section I. In oneembodiment, the invention includes an improved registration method, suchas described in U.S. Pat. No. 4,791,934, where the DRR is generated bythe fast-mapping method described herein. Another application of thefast-mapping method is the virtual fluoroscopic method now to bediscussed.

III. Virtual Fluoroscopic Imaging

In still another aspect, the invention provides a virtual fluoroscopicmethod and system for “imaging” subsurface patient tissue, e.g., thespinal region, during a surgical or diagnostic procedure on the patient.The aim of the method, in part, is to allow a surgeon to viewfluoroscopic-like or shadowgraph-like images of a patient target regionon a frequent or repeated basis, without exposing the patient and othersin the surgical station to x-radiation. For example, when doing spinalsurgery, the surgeon may need to make frequent checks on the point ofentry of a tool, or the placement of an implant, or the angle or depthof a cut, by referring to a fluoroscopic image of the region. Thepresent method allows frequent virtual fluoroscopic images to begenerated from a selected point in the surgical station.

In a preferred embodiment of the method, employing the fast-mappingalgorithm described in Section II, the surgeon can generate real-timeimages as the virtual view position and orientation is adjusted. Thisallows the surgeon, for example, to move the virtual view point aboutthe patient until an optimal fluoroscopic view of the surgical site isobtained, and then take an actual fluoroscopic image from the optimalposition and orientation.

The system for carrying out the invention generally includes thecomponents shown in FIG. 1 and in particular, a scan data file 18,computational device 20, and display device 24. As described above, thescan file consists of voxel coordinates, corresponding scan-data, e.g.,density values, for each voxel, and coordinates for registrationfeatures, e.g., selected bone features or fiducials. The scan data fileis operatively connected to the computational device, e.g., computer orprocessor, which carries out the data processing operations of thesystem to be described.

Optionally, the system may contain a tracking system, such as the OTSdescribed in Section I, which may include either or both of twouser-held instruments discussed below with reference to FIGS. 8 and 9.

As in the system described in Section II, the display device ispreferably a conventional computer color monitor having a display screencomposed of an XY array of pixels, e.g., a 1280×1024 pixel array. Themonitor provides, for each pixel, a plurality of N-bit registers,typically four 8-bit registers corresponding to blue, red, green, andalpha (gray-scale) pixel input values.

FIG. 8 illustrates a surgical setting in which a patient 12 placed on asurgical bed 14 is undergoing spinal surgery at the target regionindicated at 16. The figure shows a hand-held pointer 100 havingtracking members, such as LEDs 101, for tracking the position andorientation of the pointer in space, through a suitable sensing device(not shown) such as the OTS described above. In the setting shown, it isassumed that the surgeon wishes to find an optimal view position andorientation for a fluoroscopic image, or simply wishes to view targetstructure by fluoroscopic-like images. e.g., for purposes of surgicalplanning.

In the method, the surgeon initially selects a desired point P₁,indicated at 102, and view orientation, as at 112 in FIG. 10. This isdone, for example, by moving pointer 100 to a selected position andorientation, or by moving a frame-mounted pointer to a selected positionand orientation. Upon user-activation, the system will then generate avirtual fluoroscopic image of the target tissue from the selected pointand view orientation.

The system operates, as at 114 in FIG. 10, to generate the virtual imagesubstantially by the method described in Section II, with reference toFIGS. 3 and 4, although not necessarily employing the fast-mappingalgorithm described in Section II. Briefly, the system constructs a setof rays from the selected source point to the points in a virtualdetector array, such as the virtual detector indicated at 104, anddetermines the sum of voxel density values along each ray from the scandata. The sum along each ray is then scaled, e.g., to an 8-bit number,and these values applied to the corresponding pixels in the displaydevice, to generate a reconstructed shadowgraph image. As in Section II,the voxel coordinates may be precisely registered with the patient frameof reference, as at 118 in FIG. 10, in which case the reconstructedimage precisely matches an actual fluoroscopic image from that positionand orientation. If the scan data is not precisely registered withpatient coordinates, the reconstructed image will be approximate only asto actual position.

In a preferred embodiment, the summing method applied to the voxeldensity values employs the fast mapping method described in Section II,where the summing operations are carried out by the individual registersup to the final summing of the values in the four different pixelregisters. This method, because of its speed, allows reconstructedshadowgraph images in real time, as the user moves the pointer around inspace. This allows the surgeon to quickly and efficiently find optimalviews from which actual fluoroscopic images can be taken, or, if thescan data is in registry with patient coordinates, to view patientsubstructure accurately from continuously varying points of view, thatis, with true x-ray vision.

FIG. 8 shows selected views at points P1 (point 102) and P2 (point 106),and the virtual detector plates 104 and 108 associated with each view,as at 116 in FIG. 10. FIGS. 9A and 9B show corresponding reconstructedshadowgraph images of a patient spinal region seen from points P₁ andP₂, respectively, in accordance with the invention. Also shown in thefigures is a surgical instrument 110 employed during a surgicalposition. Sensor elements, such as LEDs 111 on the instrument, are usedfor instrument tracking, as at 120 in FIG. 10, allowing the computer toreconstruct a virtual image of the instrument from its known positionand superimpose this image on the reconstructed shadowgraph. Inparticular, with the scan data in registry with the patient coordinates,the user can view both hidden patient structure and the features of aninstrument being employed in a surgical procedure, as at 122 in FIG. 10.Alternatively, or in addition, the surgical instrument can act as apointer, to allow the surgeon to view target tissue from the positionand orientation of the tool, to augment the information provided by thevirtual fluoroscopic method.

Although the invention has been described with respect to particularcomponents, operations, and applications, it will be appreciated thatvarious modifications and changes may be made without departing from theinvention as claimed. For example, a variety of tracking system may beemployed from tracking the positions and orientations of variouspointers, instruments and recorder plates in the surgical station,without affecting the overall result or operation of the claimed methodsand systems.

What is claimed is:
 1. A method of registering volumetric scan data of apatient surgical site with actual patient position, comprising (a)obtaining, at a scan station, volumetric scan data of a surgical site ofa patient, including data relating to plural selected features in theregion, and storing the scan data in digital form in a scan-data file,(b) after moving the patient from the scan station to a surgicalstation, positioning an x-ray source and a detector screen for imagingat least a portion of said target region fluoroscopically at a selectedorientation, (c) determining the coordinates of said x-ray source anddetector at said positions in a fixed coordinate system, (d) activatingthe x-ray source to produce a fluoroscopic image of the patient regionon said detector, (e) determining the image coordinates of said selectedfeatures in the target region in said fluoroscopic image in the fixedcoordinate system, (f) using the coordinates determined in (c) and (e)to determine the actual coordinates of the selected features in saidfixed coordinate system, and (f) matching the coordinates determined in(f) with those of the same features in the scan data, to place the scandata in the frame of reference of the patient.
 2. The method of claim 1,which further includes using the scan data to construct a subsurfaceimage of the patient, as viewed from a selected position and orientationin the surgical station.
 3. The method of claim 2, wherein said selectedposition and orientation is respect to the tip of a surgical instrument,and which further includes tracking the position of said instrument inthe surgical station.
 4. The method of claim 1, wherein said pluralselected features include at least three features, and steps (b)-(e) arecarried out at different selected positions of the x-ray source anddetector screen.
 5. A system for registering volumetric scan data of asurgical site with actual patient position, comprising (a) a scan-datafile for storing, in digital form, volumetric scan data of a surgicalsite of a patient, including data relating to plural selected featuresin the region, (b) an x-ray source and a detector screen for imaging atleast a portion of said target region fluoroscopically at a selectedorientation, (c) a fixed-position sensor for determining the coordinatesof said x-ray source and detector at said positions in a fixedcoordinate system, and (d) a computational device operatively connectedto the data file, detector screen, and fixed-position sensor, for (i)determining the actual coordinates of selected features of the surgicalsite in said fixed coordinate system from the determined coordinates ofsuch features on the detector screen and from the coordinates of saidscreen, and (ii) matching the actual coordinates of said features withthose of the same features in the scan data, to place the scan data inthe frame of reference of the patient.